Modality Theory: Supporting Multimodal Interface Design

Modality theory addresses the following general problem of mapping task domain information into interactive multimodal interfaces: given any particular set of information which needs to be exchanged between user and system during task performance in context, identify the input/output modalities which constitute an optimal solution to the representation and exchange of that information. This paper proposes a research agenda for modality theory and presents two steps towards its implementation. The first is a generative taxonomy of output modalities covering the media of graphics, sound and touch. The second is a methodology for carrying out information-mapping in design practice. It is argued that modality theory may provide useful support to contemporary designers of interactive humancomputer interfaces who have begun to use a rapidly increasing number of different, and often alternative input/output modalities for the expression and exchange of information between systems and their users.